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ABSTRACT 

A  usability  study  was  used  to  measure  user  performance  and  user  preferences  for 
a  CAVE™  immersive  stereoscopic  virtual  environment  with  wand  interfaces 
compared  directly  with  a  workstation  non-stereoscopic  traditional  CAD  interface 
with  keyboard  and  mouse.  In  both  the  CAVE™  and  the  adaptable  technology 
environments,  crystal  eye  glasses  are  used  to  produce  a  stereoscopic  view.  An 
ascension  flock  of  birds  tracking  system  is  used  for  tracking  the  user ’s  head  and 
wand  pointing  device  positions  in  3D  space. 

It  is  argued  that  with  these  immersive  technologies,  including  the  use  of  gestures 
and  hand  movements,  a  more  natural  interface  in  immersive  virtual  environments 
is  possible.  Such  an  interface  cdlows  a  more  rapid  and  efficient  set  of  actions  to 
recognize  geometry,  interaction  within  a  spaticd  environment,  the  ability  to  find 
errors,  and  navigate  through  a  virtual  environment.  The  wand  interface  provides  a 
significantly  improved  means  of  interaction.  This  study  quantitatively  measures  the 
differences  in  interaction  when  compared  with  traditional  human  computer 


interfaces. 
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This  paper  provides  analysis  via  usability  study  methods  for  spatial  awareness 
termed  as  Benchmark  3.  During  testing,  testers  are  given  some  time  to  “play 
around ”  with  the  CAVE™  environment  for  familiarity  before  undertaking  a 
specific  exercise.  The  testers  are  then  instructed  regarding  tasks  to  be  completed, 
and  are  asked  to  work  quickly  without  sacrificing  accuracy.  The  research  team 
timed  each  task,  and  recorded  activity  on  evaluation  sheets  for  spatial  awareness 
Test.  At  the  completion  of  the  testing  scenario  involving  spatial  awareness,  the 
subject/testers  were  given  a  survey  document  and  asked  to  respond  by  checking 
boxes  to  communicate  their  subjective  opinions. 

Keywords:  Usability  Analysis;  CAVE™  (Cave  Automatic  Virtual  Environments);  Human 
Computer  Interface  (HCI);  Benchmark;  Virtual  Reality;  Virtual  Environments;  Competitive 
Comparison 


INTRODUCTION 

his  paper  is  an  extension  of  the  work  done  by  Satter  (2005)  on  Competitive  Usability 
Studies  of  Virtual  Environments  for  Shipbuilding.  The  key  difference  is  the  use  of  a 
new  immersive  environment  called  CAVE™.  The  significance  and  the  detail  description  of  this 
study  is  very  well  explained  by  Satter  (2012)  in  his  recent  paper.  Here  we  only  present  the  details 
of  this  usability  study.  The  CAVE™  was  developed  at  the  University  of  Illinois  at  Chicago  and 
provides  the  illusion  of  immersion  by  projecting  stereo  images  on  the  walls  and  floor  of  a  room¬ 
sized  cube.  Several  users  wearing  lightweight  stereo  glasses  can  enter  and  walk  freely  inside  the 
CAVE™.  A  head  tracking  system  continuously  adjusts  the  stereo  projection  to  the  current  position 
of  the  leading  viewer.  A  CAVE™  and  wand  system  schematic  is  shown  in  Figures  1  &  2. 


Figure  1:  Schematic  of  the  CAVE™  System  Figure  2:  The  Wand  Interface 


1.  Description 

In  order  to  evaluate  the  ability  of  each  environment/interface  to  aid  users  in  their  awareness  of  a 
design  space,  a  unique  space,  totally  unknown  to  the  users,  was  created.  For  the  test,  the  space 
created  was  a  virtual  factory  space  and  a  machine  shop.  Into  this  space  the  test  administrators 
were  able  to  inject  an  obelisk  icon  (an  elongated,  gray- white,  pyramid  topped  by  a  sphere  as  shown 
below  in  figure  3)  that  is  not  normally  found  in  any  factory  space.  Two  such  icons  were  randomly 
placed  into  the  new  space  for  each  pass  of  the  test.  From  a  common  starting  point,  users  were 
required  to  navigate  through  the  space  looking  for  the  icons  within  the  space.  The  time  required 
each  user  to  locate  each  icon  was  recorded  and  the  users  were  asked  to  note  the  location  for  each 


(placement  within  the  space). 


Figure  3:  Icon 


Upon  completion  of  the  test  each  user  was  shown  a  2-dimensional,  8.5”  x  1 1”,  plan-view  of  the 
space  and  asked  to  note  the  placement  of  each  of  the  two  icons.  The  test  administrators  then 
recorded  the  offset  (in  mm)  between  user  placement  and  the  actual  location  of  the  icons. 

This  exercise  (Benchmark  3)  was  repeated  in  each  of  the  2  environments  and  the  User  Survey 
administered  to  each  user  after  each  pass  in  each  environment.  As  with  the  other  Benchmark 
testing,  sequencing  of  the  testers  through  the  two  environments  was  randomized  so  that  not  all  of 
the  users  were  testing  the  same  interface  in  the  same  order.  This  randomization  was  used  to 


eliminate  bias  in  the  testing. 


2.  Benchmark  3,  Pass  3,  Part  1  &  2  Placement  Offsets  Analysis: 

Following  is  a  presentation  of  the  Benchmark  3,  pass  3,  part  1  and  part  2  placement  offsets  for  all 
the  users.  Pass  3  results  are  presented  here  as  representative  of  user  best-final  case  results.  All 
other  results  are  presented  in  Appendix  C  [3]. 

Figure  4  (Benchmark  3  pass  3  Icon  1  Offsets  /  B3p3-loff)  presents  user  placement  of  the  first  icon 
within  the  new  space.  The  results  clearly  indicate  a  higher  spatial  awareness  using  the  stereoscopic 
CAVE™  environment.  Using  the  stereoscopic  interface,  users  on  average  located  the  icon  within 
1 1  mm  of  its  actual  location.  User’s  placement  of  the  icon  using  the  workstation  non-stereoscopic 
environments  was  within  12.83  mm  of  its  actual  location. 

Inspection  of  the  standard  deviation  values  of  table  1  for  the  location  of  icon  1  shows  a  high 
variance  in  offset  for  the  stereoscopic  interface  and  shows  low  variance  for  the  non-stereoscopic 
interface.  This  is  an  indication  of  the  consistency  of  the  non-stereoscopic  method  in  spatial 
recognition  efforts.  Users  were  able  to  locate  the  icons  better  in  workstation  (2-dimensional  non- 
stereoscopic  environment)  on  a  2-dimensional,  8.5”  x  1 1”  paper  than  in  a  CAVE™. 


Benchmark  3  -  Pass  3  Part  1  Offsets 
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Figure  4:  B3p3-loff  Pass  3-Icon  1  Offsets 
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Table  1:  B3p3-loff  Pass  3-Icon  1  Offsets 

Figure  5  (Benchmark  3  pass  3  Icon  2  Offsets  /  B3p3-2off)  presents  user  placement  of  the  second 
icon  within  the  new  space.  The  results  clearly  indicate  a  higher  spatial  awareness  using  the 
stereoscopic  the  CAVE™  environment.  Using  the  stereoscopic  interface  users,  on  average, 
located  the  icon  within  7.77  mm  of  its  actual  location.  User  placement  of  the  icon  using  the 
workstation  non-stereoscopic  environments  was  within  13.8  mm  of  its  actual  location. 


Inspection  of  the  standard  deviation  values  of  table  2  for  the  location  of  icon  2  shows  a  high 
variance  in  offset  for  the  non- stereoscopic  interface  and  shows  low  variance  for  the  stereoscopic 
interface.  This  is  an  indication  of  the  consistency  of  the  stereoscopic  method  in  spatial  recognition 
efforts.  Users  were  able  to  locate  the  icons  much  better  in  the  CAVE™  environment  than  in  a 
workstation.  This  proves  that  users  performed  better  after  practice  in  the  CAVE™  environment 
proving  the  significance  usability  analysis. 


Benchmark  3  -  Pass  3  -  Part  2  Offsets 
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Figure  5:  B3p3-2off  Pass  3-Icon  2  Offsets 


3.  Detailed  Statistical  Analysis 


The  following  sections  present  a  detailed  statistical  analysis  of  the  Benchmark  3  results  of  the  user 
group  in  a  manner  similar  to  the  previous  Benchmarks.  As  discussed  above,  the  NCSS  software 
package  was  used  to  perform  each  analysis.  Each  set  of  user  icon  2  placement  offsets  is  first 
examined  to  determine  if  the  data  is  normally  distributed  (Gaussian  distribution)  using  the  KS 
statistic.  The  descriptive  statistics  test  results  are  presented  in  tabular  form  followed  by  the  results 
of  Levene’s  test  for  equal  variance  of  the  data.  The  null  hypothesis  (Ho)  and  alternative  hypothesis 
(Ha)  discussed  for  Benchmark  1  statistical  analysis  testing  applies  here  (Benchmark  3)  as  well. 

4.  Benchmark  3Pass  3  Statistics 

Benchmark  3,  pass  3,  icon  2  offsets  represent  each  user’s  view  of  the  placement  of  the  required 
device  in  a  foreign  space.  As  such,  the  results  of  this  pass/icon  placement  represent  a  reasonable 
characterization  of  the  user’s  spatial  awareness  within  each  environment. 

5.  B3p3-2off  -Benchmark  3  Pass  3  Descriptive  Statistics 

Table  2  presents  the  results  of  the  descriptive  statistics  analysis  of  user’s  pass  3  location  of  icon  2 
in  the  test  environment.  All  other  results  are  presented  in  Appendix  C  [3].  The  K.S.  test  is  used  to 
test  for  normality  of  data.  Since  the  P  value  is  less  than  0.1,  the  data  is  not  normal.  Next  Levene’s 
test  is  then  applied  to  test  for  equal  variance.  Since  the  P  value  is  greater  than  0.1  the  data  has  equal 
variance.  Since  the  data  is  not  normal,  Mann  Whitney  test  is  used.  A  Mann  Whitney  test  P  value 
less  than  0.1  indicates  that  medians  are  unequal  for  the  CAVE™  and  workstation.  Examination  of 
these  results  shows  that  for  the  two  environments,  the  differences  are  statistically  significant.  The 
conclusion  then  is  that  at  the  90%  confidence  level,  there  is  significant  evidence  to  support  the 


alternative  hypothesis  (Ha).  Thus,  since  the  stereoscopic  wand  environment  demonstrates  shorter 
offset  distances,  the  CAVE™  environment  is  statistically  “better”  than  non-stereoscopic 
workstation  environment  for  Benchmark  3  during  pass  3  for  Icon  2  placements. 


Mann-Whitney  Test 
Value  Pr>T  Equal? 

2.30  0.01  No 

Table  2:  B3p3-2off  Pass  3-Icon  2  Offsets 
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Benchmark  3  Pass  3  Overall  Impressions  Ratings  Analysis: 


Rat 
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Figure  6:  B3p30vr  Pass  3  Overall  Impressions  Ratings 

Figure  6  (Benchmark  3  pass  3  Overall  Impressions  Ratings  /  B3p30vr)  graphically  presents 
comparisons  of  the  Benchmark  3  (spatial  awareness)  pass  3  overall  ratings  of  the  two 
environments.  Inspection  of  the  average  ratings  shows  that  users  preferred  the  stereoscopic 
environment  (CAVE™)  over  the  non- stereoscopic  environment  (workstation). 


6.  Detailed  Statistical  Analysis 


The  following  sections  present  a  detailed  statistical  analysis  of  user  overall  impressions  ratings  of 
the  two  test  environments  following  their  3rd  and  final  pass  of  the  Benchmark  3  scenario.  All 
other  results  are  presented  in  Appendix  C[3]  .  The  statistical  analysis  of  these  ratings  provides 
insight  into  the  final  opinions  of  the  users.  As  discussed  above,  the  NCSS  software  package  was 
used  to  perform  each  analysis.  Each  set  of  user  overall  impressions  ratings  is  first  examined  to 
determine  if  the  data  are  normally  distributed  (Gaussian  distribution)  using  the  KS  statistic.  The 
descriptive  statistics  test  results  are  presented  in  tabular  form  followed  by  the  results  of  Levene’s 
test  for  equal  variance  of  the  data.  The  null  hypothesis  (Ho)  and  alternative  hypothesis  (Ha) 
discussed  for  Benchmark  land  2  statistical  analysis  testing  applies  here  (Benchmark  3)  as  well. 

7.  Benchmark  3  Pass  3  Overall  Impressions  Ratings  Statistics 

As  noted,  Benchmark  3,  pass  3,  overall  impressions  ratings  represent  each  user’s  view  of  the 
placement  of  the  required  device  in  a  foreign  space.  As  such,  these  ratings  represent  a  reasonable 
characterization  of  the  user’s  overall  impressions  of  the  interfaces  after  the  use  of  each  to  determine 
his/her  spatial  awareness  of  a  previously  unknown  environment. 

Table  3  presents  the  results  of  the  descriptive  statistics  analysis  of  user’s  Benchmark  3  pass  3 
overall  impressions  of  the  interface.  The  K.S.  test  is  used  to  test  for  normality  of  data.  Since  the 
P  value  is  less  than  0.1  for  the  CAVE™,  the  data  are  not  normal.  Levene’s  test  is  used  to  test  for 
equal  variance  and  since  the  P  value  is  greater  than  0. 1  the  data  have  equal  variance.  Since  the 
data  is  not  normal,  Mann  Whitney  test  is  used.  With  the  Mann -Whitney  test  P  value  less  than  0.1, 
which  indicates  that  the  medians  are  unequal  for  the  CAVE™  and  workstation.  Examination  of 


these  results  shows  that  for  the  two  environments,  the  differences  are  statistically  significant.  The 


conclusion  then  is  that  at  the  90%  confidence  level,  there  is  significant  evidence  to  support  the 
alternative  hypothesis  (Ha).  This  proves  that  the  CAVE™  environment  is  preferred  over 
workstation  for  Benchmark  3  pass  3  overall  impressions  subjective  ratings. 


B30P3 

#  Users 

Mean 

St.  Dev. 

Low 

High 

P  Value 

Normal? 

C  V 

Cave 

30 

4.69 

0.2 

4.2 

4.90 

<0.10 

No 

4.00% 

W/S 

30 

4.33 

0.16 

4 

4.75 

>0.10 

Yes 

4.00% 

Homogeneity  of  Variance 

Test  for  Differences 

Levene's  Test 

Equal 

Var? 

Mann-WhitneyTest 

Equal? 

Significant? 

F-Value 

P  Value 

Value 

P  Value 

Cave  vs  W/S 

2 

0.16 

Yes 

-5.31 

<0.001 

No 

Cave 

Table  3:  B3p30vr  Pass  3  Overall  Impressions  Ratings  Descriptive  Statistics 


8.  Benchmark  3  Pass-to-Pass  Comparison  Analysis: 

B3  Parti  Pass  to  Pass  Comparison 


Passl  to  Pass2 

Pass2  to  Pass  3 

Passl  to  Pass3 

Diff 

% 

Diff 

% 

Diff 

% 

Cave 

12.5 

51% 

1.03 

9% 

11.7 

48% 

W/S 

14.7 

49% 

2.54 

17% 

17.27 

57% 

Table  4:  B3I1  pass-to-pass  Comparison  of  Offset  distances 

Table  4  presents  pass-to-pass  comparison  of  Benchmark  3  part  1/Icon  1  offsets.  The  positive  values 
in  table  4  prove  that  pass  1  offsets  were  greater  than  pass  2  and  pass  2  offsets  were  greater  than 
pass  3.  This  proves  that  user’s  placement  of  the  icon  on  the  paper  improved  from  pass-to-pass  with 
respect  to  the  icon’s  exact  location  in  the  two  test  environments.  For  example  a  value  of  57%  for 
Workstation  (pass  1  to  pass  3)  is  calculated  as  (30.1-12.83)/30.1,  where  30.1  and  12.83  represent 
the  means  of  Benchmark  3  part  1/Icon  1  offsets  for  pass  1  and  pass  3  respectively.  From  table  4 


one  can  conclude  that  user’s  showed  more  improvement  from  pass  to  pass  in  workstation  than  in 
CAVE™.  This  is  due  to  the  fact  that  users  were  able  to  place  the  icons  better  in  workstation  (2- 
dimensional  non-stereoscopic  environment)  on  a  2-dimensional,  8.5”  x  11”  paper  than  in  a 
CAVE™. 


B3  Part2  Pass  to  Pass  Comparison 


Passl  to  Pass2 

Pass2  to  Pass  3 

Passl  to  Pass3 

Diff 

% 

Diff 

% 

Diff 

% 

Cave 

4.5 

35% 

0.53 

6% 

5.03 

39% 

W/S 

11 

39% 

3.7 

21% 

14.6 

51% 

Table  5:  B3I2  pass-to-pass  Comparison  of  Offset  distances 

Table  5  (Benchmark  3  Icon  2  or  part  2  pass-to-pass  comparison  /  B3I2)  presents  pass-to-pass 
comparison  of  Benchmark  3  part  2/Icon  2  offsets.  The  positive  values  in  table  5  prove  that  pass  1 
offsets  were  greater  than  pass  2  and  pass  2  offsets  were  greater  than  pass  3.  For  example  a  value 
of  51%  for  Workstation  (pass  1  to  pass  3)  is  calculated  as  (28. 53-13. 8)/28.53,  where  28.53  and 
13.8  represent  the  means  of  Benchmark  3  part  2/Icon  2  offsets  for  pass  1  and  pass  3  respectively. 
From  table  5  one  can  conclude  that  user’s  showed  more  improvement  from  pass  to  pass  in 
workstation  than  in  the  CAVE™.  This  is  due  to  the  fact  that  users  were  able  to  place  the  icons 
better  in  workstation  (2-dimensional  non-stereoscopic  environment)  on  a  2-dimensional,  8.5”  x 
11”  paper  than  in  a  CAVE™. 


B3  Overall 

Ratings  Pass  to  Pass  Comparison 

Passl  to  Pass2 

Pass2  to  Pass  3 

Passl  to  Pass3 

Diff 

% 

Diff 

% 

Diff 

% 

Cave 

-0.82 

-23% 

-0.33 

-8% 

-1.15 

-32% 

W/S 

-0.24 

7% 

-0.71 

-20% 

-0.95 

-28% 

Table  6:  B3  Overall  Impressions  Ratings  pass  to  pass  Comparison 


Table  6  presents  pass-to-pass  comparison  of  Benchmark  3  overall  impressions  subjective  ratings. 
The  negative  values  in  table  6  prove  that  pass  1  ratings  were  lower  than  pass  2  and  pass  2  ratings 
were  lower  than  pass  3.  For  example  a  value  of  -28%  for  Workstation  (pass  1  to  pass  3)  is 
calculated  as  (3.38-4.33)/3.38,  where  3.38  and  4.33  represent  the  means  of  Benchmark  3  overall 
impressions  ratings  for  pass  1  and  pass  3  respectively.  From  table  6  one  can  conclude  that  the 
CAVE™  is  preferred  over  workstation. 


Usability  Survey 

User  ID: 

Environment:  19"  CAD  Stereo 

Pass:  1  2  3 

Very 

Good 

S 

Good 

4 

Neutral 

3 

Poor 

2 

Very 

Poor 

1 

Navigation 

1 

Initial  impression  of  navigational  modes 

2 

Gross  control  movement 

3 

Speed  of  cursorfpomter  movement 

4 

Ability  to  make  fine  adiustments  to  the  placement  of  the  corsor/pomter 

5 

Ability  to  recover  cursor /pointer  movements 

6 

Ease  of  use 

7 

After-test  repression  of  the  navigational  modes 

Locating 

1 

Initial  impression  of  the  interface  in  locating  specific  parts/equipment 

2 

Ease  of  identification  of  selected  part/equipment 

3 

Ability  to  make  fine  adiustments  «i  selecting  specific  parts/eguipment 

4 

Ease  of  use 

5 

After-test  impression  of  the  location; selection  mechanism 

Movement 

1 

Intial  impression  of  the  interface  for  relocating  parts/eguipment 

2 

Ease  of  movement  across  the  three  axis 

3 

Ability  to  make  fine  pa  rt/eo  moment  movement  adiustments 

4 

Ease  of  use 

5 

After-test  repression  of  the  movement  mechanism 

General 

1 

Ineial  impression  of  the  overall  system 

2 

Ability  to  relate  a  2D  planform  to  the  space  as  presented 

3 

'Intuitiveness"  of  the  interface  ■  do  the  controls  foaovr  expected  use’ 

4 

Overall  ease  of  use 

5 

After-test  repression  of  the  overall  system 

Comments 

Figure  7:  Usability  Survey  Questionnaire  (Satter,  2005) 


9.  CONCLUSIONS 


For  Benchmark  3  (spatial  awareness)  the  statistics  shows  better  results  (lower  offset  distances  and 
higher  subjective  ratings)  for  the  CAVE™  in  both  objective  and  subjective  measures  than  the 
workstation.  We  prove  that  the  CAVE™  is  preferred  over  workstation  by  users. 
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